Method and System for  Automatically Mapping Fluid Objects on a Substrate

ABSTRACT

A method and system for mapping fluid objects on a. substrate using a microscope inspection system that includes a light source, imaging device, stage for moving a substrate disposed on the stage, and a control module. A computer analysis system includes an object identification module that identifies for each of the objects on the substrate, an object position on the substrate including a set of X, Y, and θ coordinates using algorithms, networks, machines and systems including artificial intelligence and image processing algorithms. At least one of the objects is fluid and has shifted from a prior position or deformed from a prior size.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.16/164,990, tiled Oct. 19, 2018, the entire content of which isincorporated herein by reference.

TECHNICAL FIELD

The embodiments of the present disclosure relate to automaticallymapping fluid objects on a substrate.

BACKGROUND

Inspecting materials for uniformity and detection of anomalies isimportant in disciplines ranging from manufacturing to science tobiology. Inspection often employs microscope inspection systems toexamine and measure electronic objects on a substrate (e.g., a wafer) orfeatures of a biological specimen mounted on a slide. Specimens asunderstood by a person of ordinary skill in the art refer to an articleof examination (e.g., a wafer or a biological slide). Electronic objectson a substrate can include devices such as transistors, resistors,capacitors, integrated circuits, microchips, etc. Biological specimensare typically mounted on slides for microscopic inspection. Objects, asunderstood by a person of ordinary skill in the art, has a broad meaningas provided in the specification and recited in the claims, and canrefer to electronic objects on a substrate or biological objects such ascells, tissue or the like found in a biological specimen mounted on aslide among others. Although the following description refers toexamining objects on a substrate that are electrical in nature, theautomatic mapping described herein can be used to examine biologicalspecimens and objects mounted on slides.

Microscope inspection systems can be used to image objects on asubstrate for later analysis. To facilitate accurate analysis, it ishelpful to capture consistent images of like objects, or consistentimages of an object and its reference template (sometimes referred to asa golden template). For example, if an object is smaller than the fieldof view of an imaging device, then like objects can be aligned in thesame way in relation to an imaging device, so that captured images ofthe like objects all show similar alignment of the imaged object(referred to herein as “imaging alignment position”). In one embodiment,as shown for example in FIGS. 1A and 1B, the upper left corner 115, 115of objects 120, 120 each appears in the upper left corner of the fieldof view of an imaging device, represented by a single square 110, 110.Although the orientation of object 120 has rotated, field of view 110has rotated as well, to maintain the same alignment of objects 120 and120 in the captured images.

Note, the term field of view as understood by a person of ordinary skillin the art is in the context of digital microscope and refers to an areaof examination that is captured at once by an image sensor. Further, aperson of ordinary skill in the art will readily understand that theterms field of view, image and tile are used interchangeable herein.

In another example, as shown in FIG. 2, when an object 220 on substrate310 exceeds the field of view of an imaging device, as represented byeach tile 215, then a sequence of images (e.g., tiles 1-18) might beneeded to capture the entire object. Note, field of view and tile areused interchangeably herein. To facilitate accurate analysis, it ishelpful to capture the sequence. of images in a consistent manner, witha similar imaging alignment position, across like objects or compared toa reference template. In one example, a first image can be capturedstarting at a specific feature on the object or at a specific location(e.g., upper left corner 115) on the object (referred to herein as the“starting scan position.” and indicated by *) and subsequent. images canbe captured, for example, in a predefined sequencing path (e.g., in aserpentine manner as indicated by sequencing path 230 as shown on FIG.2). Each image in the sequence can be assigned a number (e.g., 1-18) andimages with the same number can be compared across like objects or to areference template.

Knowing the exact position and orientation of each object and/orfeatures of the objects on a substrate can facilitate correct alignmentof a stage, imaging device and object to capture images where likeobjects are consistently aligned within the field of view, or a similarsequence of images are captured for like objects. Aside from imagecapture, knowing the position and orientation of an object and/orfeature of an object on a substrate can be useful for various stages ofa manufacturing or an examination process and/or for anomaly analysis.In some embodiments, an object can have indicators on the object itselfto help determine the orientation of the object (e.g., asterisk (*) 225a that appears in the upper left corner and plus sign (+) 225 b thatappears in the lower right corner of specimen 220).

An initial object layout map can specify the X, Y, θ coordinates of eachobject on a substrate (“expected position” or “original position”). Forexample, X, Y can refer to a coordinate position of each object 220 inrelation to a common reference point on a substrate (e.g., an originpoint), and θ can refer to the orientation of each object 220 or abiological specimen in relation to an origin point relative to a known.coordinate system, as explained further within. However, an initialobject layout map typically does not account for movement (i.e.,movement is referred to in the specification and claims as “fluid” andmeans that an object is capable of movement from an original position toa later position) of the objects during examination and/or manufacturingprocess from their initial X, Y, θ coordinates. When printing objects ona bendable or elastomeric (“flexible”) substrate (e.g., polyimide, PEEKor transparent conductive polyester film), printing a flexible object(e.g., a flexible OLED), examining fluid biological specimens mounted ona slide and/or examining objects post-dicing (e.g., on a hoop ring,Gel-Pak®, waffle pack), the objects can shift from their original orexpected X, Y, θ coordinates (e.g., as specified in an initial objectlayout map). Deformation of a flexible substrate and/or flexible objectcan also occur, which can also alter the expected X, Y, θ coordinates ofobjects on a substrate or biological specimens on a slide. Deformation(also known as morphing by a person of ordinary skill in the art) canrefer to deviations between an object and a reference object in overalldimensions and/or individual features of the objects. The referenceobject can refer to a reference template image for that object typeand/or an earlier version of the object.

Accordingly, it is desirable to provide new mechanisms for automaticallymapping fluid objects on a substrate (e.g., by determining the X, Y, θcoordinates of each object on a substrate) to locate objects that. haveshifted from their expected coordinates on a substrate (e.g., ascompared to an initial object layout map), as well as to predict X, Y, θcoordinates of an object on a substrate at different stages in anexamination or manufacturing process.

SUMMARY OF THE EMBODIMENTS OF THE PRESENT DISCLOSURE

An embodiment of the present disclosure is directed to a method formapping fluid objects on a substrate using a microscope inspectionsystem that includes a microscope inspection system having a microscopesystem and a computer analysis system. The microscope system includes alight source, imaging device, stage for moving a substrate disposed onthe stage, and a control module. The computer analysis system includesan object identification module. The method includes the steps ofperforming a scan of the substrate using the microscope inspectionsystem and identifying, for each of the objects on the substrate, anobject position on the substrate including a set of X, Y, and θcoordinates using algorithms, networks, machines and systems includingartificial intelligence and image processing algorithms. At least one ofthe objects is fluid and has shifted from a prior position or deformedfrom a prior size. The method also includes a step of generating objectmapping information that reflects the position of each of the objectsand a shift or deformity amount for each of the objects. The step ofgenerating object mapping information is done automatically usingalgorithms, networks, machines and systems including artificialintelligence and image processing algorithms.

Another embodiment of the present disclosure is directed to a method formapping fluid objects on a substrate using a microscope inspectionsystem that includes a microscope inspection system having a microscopesystem and a computer analysis system. The microscope system includes alight source, imaging device, stage for moving a substrate disposed onthe stage, and a control module. The computer analysis system includesan object identification module and an object layout prediction module.The method includes the steps of performing a scan of the substrateusing the microscope inspection system and identifying, for each of theobjects on the substrate, an object position on the substrate includinga set of X, Y, and θ coordinates using algorithms, networks, machinesand systems including artificial intelligence and image processingalgorithms. At least one of the objects is fluid and has shifted from aprior position or deformed from a prior size. The method also includes astep of generating object mapping information that reflects the positionof each of the objects and a shift or deformity amount for each of theobjects. The step of generating object mapping information is doneautomatically using algorithms, networks, machines and systems includingartificial intelligence and image processing algorithms.

Yet another embodiment of the present disclosure is directed to a systemfor mapping fluid objects on a substrate that includes a microscopesystem having a light source, imaging device, stage for moving asubstrate disposed on the stage, and a control module. The imagingdevice scans the substrate. The system also includes an object layoutidentification module for identifying for each of the objects on thesubstrate, an object position on the substrate including a set of X, Y,and θ coordinates using algorithms, networks, machines and systemsincluding artificial intelligence and image processing algorithms. Atleast one of the objects is fluid and at least one of the objects hasshifted from a prior position or deformed from a prior size. The objectlayout identification module generates object mapping information thatreflects the position of each of the objects and a shift or deformityamount for each of the objects.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the principles briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only exemplary embodiments of the disclosure and are nottherefore to be considered to be in their scope, the principles hereinare described and explained with additional specificity and detailthrough the use of the accompanying drawings in which:

FIG. 1A shows a top view of a field of view of an object on an imagingtile.

FIG. 1B shows a top view of a field of view of an object on an imagingtile, where the field of view and the object have been rotated ascompared to FIG. 1.

FIG. 2 shows an object on a substrate that exceeds the field of view ofan imaging device, as represented by each tile, then a sequence ofimages (e.g., tiles 1-18) is needed to capture the entire object.

FIG. 3A shows an initial object layout which includes example layouts ofobjects A-F on a substrate at different stages in a manufacturingprocess or occurring at different times in an examination process.

FIG. 3B shows a later object layout which includes example layouts ofobjects A-F on a substrate 310 at different positions than those shownin FIG. 3A at different stages in a manufacturing process or occurringat different times in an examination process.

FIG. 4 shows an example of an automatic mapping microscope inspectionsystem including a microscope system and computer analysis module.

FIG. 5A shows a side view of an embodiment of a microscope systemshowing an imaging device, light source, objectives, specimen, stage,control module, and computer analysis module.

FIG. 5B shows a front view of an embodiment of a microscope systemshowing an imaging device, light source, objective, specimen, stage,control module, and computer analysis system.

FIG. 6A shows example method steps for automatically mapping a fluidobject on a substrate.

FIG. 6B shows example method steps for automatically mapping a fluidobject on a substrate including a prediction step.

FIG. 7A shows object Q at a first instance with object Q aligned in anupper left portion of a virtual tile.

FIG. 79 shows object Q at a second instance, after it has shifted from afirst position as shown in FIG. 7A.

FIG. 8 shows the general configuration of an embodiment of a computeranalysis system.

FIG. 9 shows an image processing algorithm that is first trained withtraining data so that the object identification module can detect andrecognize objects on a substrate to provide a certain output.

FIGS. 10A and 10B show an example embodiment where orientation iscalculated based on the rotation of an object A from a first point intime (as represented by FIG. 10A) to a second point in time (asrepresented by FIG. 10B), using the same two reference points on objectA: A₁ and A₂.

FIG. 11 shows an example training model that uses certain inputs andoutputs to feed into a specimen layout algorithm to obtain a certainhypothesis

DETAILED DESCRIPTION

In accordance with some embodiments of the disclosed subject matter,mechanisms (which can include methods, systems, devices, apparatuses,etc.) for automatically mapping fluid objects on a substrate (e.g., bydetermining the X, Y, θ coordinates of each object on a substrate) tolocate objects that have shifted from their initial or expected positionon a substrate (e.g., as compared to an initial object layout map) areprovided. This can be useful to facilitate alignment of a stage, imagingdevice and object to capture suitable image(s) of an object, tocalibrate later stages of an examining or manufacturing process toaccount for any shifting of objects and/or for anomaly analysis. Thiscan also be useful to locate objects when an initial object layout mapfor a substrate is not provided, even when the objects have not shiftedat all. In some embodiments, automatic mapping includes, not the actualX, Y, θ coordinates objects on a substrate, but predicting where the X,Y, θ coordinates of objects on substrate will be during different stagesof an examining or manufacturing process. This can be useful toappropriately position objects on a substrate or to calibrate stepsand/or components in a manufacturing or an examining process toaccommodate expected shifting and/or deformation of an object and/orsubstrate.

FIGS. 3A (an initial object layout) and 3B (a later object layout),illustrate example. layouts of objects A-F on a substrate 310 atdifferent stages in a manufacturing process or occurring at differenttimes in an examining process, in accordance with some embodiments ofthe disclosed subject matter. Each tile 215 represents an image or afield of view. As shown in FIGS. 3A and 3B, a Cartesian XY coordinatesystem can be used to define the X, Y location of each object A-F onsubstrate 310. The XY coordinate location of each object A-F representsa distance from coordinate axes 312A and 312B that meet at origin point(O). In some embodiments, the coordinate axes can be a pair ofperpendicular lines that extend from two reference indices 313A and 313Bfound on substrate 310. Note that coordinate axes 312A and 31213 andorigin point O are just examples, the coordinate location of an object.A-F can be measured from other coordinate axes and origin point O and/orfrom another reference point(s). In other embodiments, an object can belocated by: its polar coordinates in relation to an origin point and/orany other suitable location. The XY location can refer to the locationof a specific portion of an object (e.g., upper left hand corner) and/orthe location of a specific feature of an object.

Each object A-F includes orientation marks 314A (e.g., an asterisk (*))and 314B (e.g., a plus sign (+)) that can be used to determine theorientation of an object in relation to origin point O. For example, inan initial layout of the objects as shown in FIG. 3A, the asteriskappears in the upper left corner of each object and the plus signappears in the lower right corner of each object. FIG. 3A represents amodel position for objects A-F. The model position can also be the sameposition as a reference template for objects A-F and used to analyze anyshift in object A-F, as shown in FIG. 3B. In FIG. 3B, many of theobjects have shifted from their initial XY locations, and theorientation of the objects has changed as well, as demonstrated by thenew location of the asterisk and plus signs in relation to origin pointO. Orientation marks 314B, 314B are just examples and other orientationmarks can be used to determine orientation and degree of rotation froman expected or an initial orientation. In some embodiments, features ofthe object can be used to determine orientation of the object. Note,objects on a substrate can be the same type of object or different typesof objects.

As disclosed herein, in some embodiments, artificial intelligence can beused to detect an object, classify an object type, identify an imagingalignment position for an object, identify a starting scan position,determine the X, Y, and θ coordinates of an object and/or predict the X,Y, θ coordinates of each object on a substrate. The artificialintelligence algorithms can include one or more of the following, aloneor in combination: machine learning, hidden Markov models; recurrentneural networks; convolutional neural networks; Bayesian symbolicmethods; general adversarial networks; support vector machines; imageregistration methods; applicable machine learning techniques; applicablerule-based system; and/or any other suitable artificial intelligencealgorithm. Such algorithms, networks, machines and systems provideexamples of structures used with respect to any “means for automaticallydetecting an object using artificial intelligence.”

FIG. 4 illustrates an example automatic mapping microscope inspectionsystem 400 that can implement automatically mapping fluid objects on asubstrate, according to some embodiments of the disclosed subjectmatter. Automatically mapping fluid objects on a substrate can includefor each object on a substrate (or a subset of objects): i) detectingand classifying an object; ii) determining the X, Y, θ coordinates of anobject on a substrate; iii) determining object deformation; iv)determining object shifting; v) determining an object starting scanposition; vi) determining an object imaging alignment position; and/orvii) determining an object sequencing path.

At a high level, the basic components of an automatic mapping microscopeinspection system 400, according to some embodiments, include microscopesystem 410 and a computer analysis system 450. The functionality ofcomputer analysis system 450 can be incorporated into microscope system410 (as shown, for example, in FIGS. 5A and 5B) or can be a separatecomponent (as shown for example in FIG. 4). Microscope system 410 caninclude an illumination source 415 to provide light to an object, animaging device 420, a stage 425, a low-resolution objective 430, a highresolution objective 435, and control module 440 comprising hardware,software and/or firmware.

Microscope system 410 can be implemented as part of any suitable type ofmicroscope. For example, in some embodiments, system 410 can beimplemented as part of an optical microscope that. uses transmittedlight or reflected light. More particularly, system 410 can beimplemented as part of the nSpec® optical microscope available fromNanotronics Imaging, Inc. of Cuyahoga Falls, Ohio. Microscope system 410can also be implemented as part confocal or two-photon excitationmicroscopy.

FIGS. 5A (side view) and 5B (front view), show the general configurationof an embodiment of microscope system 410, in accordance with someembodiments of the disclosed subject matter. According to someembodiments, microscope system 410 can include low resolution objective430 and high resolution objective 435. Low resolution objective 430 andhigh resolution objective 435 have different resolving powers. Lowresolution objective 430 and high resolution objective 435 can also havedifferent magnification powers, and/or be configured to operate withbright field/dark field microscopy, differential interference contrast(DIC) microscopy and/or any other suitable form of microscopy includingfluorescence. In some embodiments, high resolution scanning of an objectcan be performed by using a high resolution microscope like a scanningelectron microscope (SEM), a transmission electron microscope (TEM),and/or an atomic force microscope (AFM). In some embodiments, a highresolution microscope can be a microscope that has a magnifying power(e.g., 100×) greater than a low resolution microscope (e.g., 5×). Theobjective and/or microscope technique used to inspect an object can becontrolled by software, hardware, and/or firmware in some embodiments.

In some embodiments, an XY translation stage can be used for stage 425.The XY translation stage can be driven by stepper motor, server motor,linear motor, piezo motor, and/or any other suitable mechanism. The XYtranslation stage can be configured to move an object in the X axisand/or Y axis directions under the control of any suitable controller,in some embodiments. An actuator (not shown but known in the art) can beused to make coarse focus adjustments of, for example, 0 to 5 mm, 0 to10 mm, 0 to 30 mm, and/or any other suitable range(s) of distances. Anactuator can also be used in some embodiments to provide fine focus of,for example, 0 to 50 μm, 0 to 100 μm, 0 to 200 μm, and/or any othersuitable range(s) of distances. In some embodiments, microscope system410 can include a focus mechanism that adjusts stage 425 in a Zdirection towards and away from objectives 430 and 435 and/or adjustslow resolution objective 430 and high resolution objective 435 towardsand away from stage 425.

Light source 417 can vary by intensity, number of light sources used,and/or the position and angle of illumination. Light source 417 cantransmit light through reflected light illuminator 418 and can be usedto illuminate a portion of a specimen, so that light is reflected upthrough tube lens 423 to imagine device 420 (e.g., camera), and imagingdevice 420 can capture images and/or video of the object. In someembodiments, the light source 417 used can be a white light collimatedlight-emitting diode (LED), an ultraviolet collimated LED, lasers orfluorescent light.

In some embodiments, imaging device 420 can be a rotatable camera thatincludes an image sensor. The image sensor can be, for example, a CCD, aCMOS image sensor, and/or any other suitable electronic device thatconverts light into one or more electrical signals. Such electricalsignals can be used to form images and/or video of an object. Someexample methods for rotating a camera that can be used by microscopesystem 410 are described in U.S. Pat. No. 10,048,477 entitled “Cameraand Object Alignment to Facilitate Large Area Imaging in Microscopy,”which is hereby incorporated by reference herein in its entirety.

Different topographical imaging techniques can be used (including butnot limited to, shape-from-locus algorithms, shape-from-shadingalgorithms, photometric stereo algorithms, and Fourier ptychographymodulation algorithms) with a predefined size, number, and position ofilluminating light to generate one or more three-dimensional topographyimages of an object.

In some embodiments, control module 440 as shown in FIG. 5a , includes acontroller and controller interface, and can control any settings ofautomatic mapping microscope inspection system 400 (e.g., light source417, low resolution objective 430 and high resolution objective 435,stage 425, and imaging device 420), as well as communications,operations (e.g., taking images, turning on and off a light source 417,moving stage 425 and/or objectives 430, 435, and/or rotating imagingdevice 420). Control module 440 and applicable computing systems andcomponents described herein can include any suitable hardware (which canexecute software in some embodiments), such as, for example, computers,microprocessors, microcontrollers, application specific integratedcircuits (ASICs), field-programmable gate arrays (FPGAs) and digitalsignal processors (DSPs) (any of which can be referred to as a hardwareprocessor), encoders, circuitry to read encoders, memory devices(including one or more EPROMS, one or more EEPROMs, dynamic randomaccess memory (“DRAM”), static random access memory (“SRAM”), and/orflash memory), and/or any other suitable hardware elements. In someembodiments, individual components within automatic mapping microscopeinspection system 400 can include their own software, firmware, and/orhardware to control the individual components and communicate with othercomponents in automatic mapping microscope inspection system 400.

In some embodiments, communication between the control module (e.g., thecontroller and controller interface) and the components of automaticmapping microscope inspection system 400 can use any suitablecommunication technologies, such as analog technologies (e.g., relaylogic), digital technologies (e.g., RS232, ethernet, or wireless),network technologies (e.g., local area network (LAN), a wide areanetwork (WAN), the Internet) Bluetooth technologies, Near-fieldcommunication technologies, Secure RF technologies, and/or any othersuitable communication technologies.

In some embodiments, operator inputs can be communicated to controlmodule 440 using any suitable input device (e.g., keyboard, mouse,joystick, touch).

Referring back to FIG. 4, computer analysis system 450 of automaticmapping microscope inspection system 400 can be coupled to, or includedin, microscope system 410 in any suitable manner using any suitablecommunication technology, such as analog technologies (e.g., relaylogic), digital technologies (e.g., RS232, ethernet, or wireless),network technologies (e.g., local area network (LAN), a wide areanetwork (WAN), the Internet) Bluetooth technologies, Near-fieldcommunication technologies. Secure RF technologies, and/or any othersuitable communication technologies. Computer analysis system 450, andthe modules within computer analysis system 450, can be configured toperform a number of functions described further herein using imagesoutput by microscope system 410 and/or stored by computer readablemedia.

Computer analysis system 450 can include any suitable hardware (whichcan execute software in some embodiments), such as, for example,computers, microprocessors, microcontrollers, application specificintegrated circuits (ASICs), field-programmable gate arrays (FPGAs), anddigital signal processors (DSPs) (any of which can be referred to as ahardware processor), encoders, circuitry to read encoders, memorydevices (including one or more EPROMS, one or more EEPROMs, dynamicrandom access memory (“DRAM”), static random access memory (“SRAM”),and/or flash memory), and/or any other suitable hardware elements.

Computer-readable media can be any available media that can be accessedby the computer and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media can comprise computer storage mediaand communication. media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digital videodisk (DVD) or other optical disk storage, magnetic cassettes, magnetictape, magnetic disk storage or other magnetic storage devices, or anyother medium which can be used to store the desired information andwhich can be accessed by the computer.

According to some embodiments, computer analysis system 450 can includean object identification module 460 (described later and shown in FIG.8) and an object layout prediction module 470 (described later and shownin FIG. 8.)

FIGS. 6A and 6B show at a high level, example methods 600A and 600B forautomatically mapping a fluid object on a substrate using artificialintelligence, in accordance with some embodiments of the disclosedsubject matter. In some embodiments, automatic mapping operation 600Aand 600B can use automatic mapping microscope inspection system 400.Further details explaining how each module of computer analysis system450 can be configured, in accordance with some embodiments of thedisclosed subject matter, will be described in connection with FIG. 8.

At 610A, microscope system 410 can scan a substrate using for examplelow resolution objective 430 or high resolution objective 435. In someembodiments, the substrate can be scanned by moving imaging device 420and/or stage 425 in an X/Y direction until the entire surface or adesired area. (“region of interest”) of a substrate is scanned. A lowresolution scan can refer to a series of low resolution images of asubstrate, or a portion of a substrate, captured and generated byimaging device 420 using low resolution objective 430. A high resolutionscan can refer to a series of high resolution images, of a substrate, ora portion of a substrate, captured and generated by imaging device 420using high resolution objective 435.

In some embodiments, each image of a substrate is referred to as a tile215 (as shown in FIG. 2), wherein each tile 215 can be located by its XYcoordinate position in a substrate space. The tiles 215 can be stitchedtogether based on their XY coordinate positions and/or feature-basedregistration. methods into a single coherent scan of the substrate 310.In some embodiments, one or more areas of a substrate 310 can be scannedby using different focus levels and moving stage 425 and/or lowresolution objective 430 in a Z direction. Referring back to FIG. 6A, at620A, object identification module 460 (shown in FIG. 8), can receivelow or high resolution images of the scanned substrate and useartificial intelligence algorithms, computer vision and/or othersuitable computer programs (as explained further herein) to determineand generate object mapping information including: i) detecting objectson a substrate; ii) identifying object type; iii) determining anobject's X, Y, θ coordinates; iv) identifying an imaging alignmentposition and determining the X, Y, θ coordinates of such position; v)identifying a starting scan position and determining the X, Y, θcoordinates of such position; vi) determining an imaging sequence foreach object, or any number of objects, on a substrate; vii) calculatingobject and/or substrate deformation as compared to a prior or originalsize) and/or object shift from an original position (e.g., as comparedto an initial object layout map). In some embodiments, object mappinginformation can be used by object layout identification module 460 togenerate an object layout map of the substrate that represents a currentlayout of the objects on a substrate, which due to object and/orsubstrate flexibility can be different from an initial object layout mapof the substrate.

Referring again to FIG. 6A, at 630A, object identification module 460,can transmit object mapping information to control module 440. In someembodiments (e.g., for embodiments where the object is smaller than thefield of view), control module 440 can align the stage, the imagingdevice and the object to account for an object's new X, Y, θ coordinatesand/or deformity. For example, if the object is smaller than the fieldof view, then control module 440 can drive the stage 425 and/or imagingdevice 420 to an imaging alignment position for each object (or anynumber of objects) on the substrate, in accordance with the objectmapping information for the substrate. The imaging alignment positioncan be based on a representative position of a similar object to or areference template for the object being scanned. In other embodiments(e.g., if the object is larger than the field of view), control module440 can drive the stage 425 and/or the imaging device 420, to a startingscan position for each object (or any number of objects) on thesubstrate to take a sequence of images in accordance with the imagingsequence for that object type and the object mapping information for theobject. The starting scan position and imaging sequence can be based ona representative position and sequence of a similar object to or areference template for the object being scanned. In some embodiments,the type of object that is being imaged will determine the starting scanposition, imaging alignment position and/or imaging sequence. In furtherembodiments, a sequence of images for an object can be pre-processedinto one larger image and compared to a reference object for analysis.

In further embodiments, based on the X, Y, θ coordinates and/ordeformity of an object a virtual tile 710 (shown in FIG. 7A) can becreated that encompasses the dimensions of an object. All objects of thesame type can be aligned within a virtual tile in the same way tofacilitate analysis. An object can be aligned within a virtual tile,similar to the alignment of a reference template for that object. Avirtual tile is a number of smaller scans that are pre-processed intoone larger scan. FIGS. 7A shows object Q at a first instance with objectQ aligned in an upper left portion of virtual tile 710. FIG. 7B showsobject Q at a second instance, after it has shifted from a firstposition. Virtual tile 710 is similarly shifted to encompass object Q atits new position so that it is similarly aligned in the upper leftportion of the tile. Virtual tile 710 can be drawn at the pixel level ofa scan or at the tile level (i.e., field of view). A person of ordinaryskill in the art will readily understand that a virtual tile can becreated regardless of whether any preliminary stage, imaging device, orspecimen alignment was performed.

In FIG. 6B, steps 610B and 620B can be the same as described inconnection with FIG. 6A, steps 610A and 620A, respectively. Further, at620B, object layout identification module 460 can compare the generatedobject layout map of objects on a substrate to an initial or priorobject layout map of objects on a substrate and generate feedback data.The feedback data can include, but is not limited to, the amount thatthe dimensions of an object has changed compared to the dimensions of areference image of the object. A reference image can include a referencetemplate for that type of object and/or an earlier version of the objectitself. The feedback data can also include the amount that an object hasshifted from its initial or earlier position and/or a change inorientation of an object on a substrate.

At step 630B of FIG. 6B, object identification module 460, can transmitobject mapping information and feedback data to object layout predictionmodule 470. Object layout prediction module 470 can use the objectmapping information and feedback data from object identification module460, in combination with other information about a substrate and theobjects on the substrate, to predict the X, Y, θ coordinates of objects(e.g., orientation and shift) on a substrate for a particular stage of amanufacturing and/or examining process and/or to predict, objectdeformity. In some embodiments, object layout prediction module 470 cancompare the predicted position to the actual position informationgenerated by object identification module 460. If the actual X, Y, θcoordinates of an object exceeds a predefined tolerance for the type ofobject and/or substrate being inspected (e.g., the shift in objectposition was much greater than predicted or the average shift in objectposition for the substrate was much greater than expected), then objectlayout prediction module 470 can generate an alert. Similarly, objectlayout prediction module 470 can compare the predicted object deformityto the actual object deformity information generated by objectidentification module 460. If the actual object deformity exceeds apredefined tolerance for the type of object and/or specimen beinginspected (e.g., the deformity for the object or the average objectdeformation on the substrate was much greater than predicted), thenobject layout prediction module 470 can generate an alert.

In some embodiments, object layout prediction module 470 can compare thepredicted object position to the actual position information generatedby object identification module 460 and/or compare the predicted objectdeformity to the actual object deformity information generated by objectidentification module 460 to assess the accuracy of the predictions ofobject layout prediction module 470. In some embodiments, if theaccuracy of the predictions of object layout prediction module 470 meetsa predefined tolerance, for a predefined time (e.g., when object layoutprediction module 470 is sufficiently trained), then steps 610B and 620Bcan be omitted. The information generated by object layout predictionmodule 470 can be transmitted directly to control module 440, which canalign stage 425, imaging device 420 and a specimen to account for anobject's new X, Y, θ coordinates and/or deformity, as discussed inconnection with step 630A of FIG. 6A.

The division of when the particular portions of automatic mappingoperation 600A and 600B are performed can vary, and no division or adifferent division is within the scope of the subject matter disclosedherein. Note that, in some embodiments, blocks of automatic mappingoperation 600A and 600B can be performed at any suitable times. Itshould be understood that at least some of the portions of automaticmapping operation 600A and 600B described herein can be performed in anyorder or sequence not limited to the order and sequence shown in anddescribed in connection with FIGS. 6A and 6B, in some embodiments. Also,some portions of process 600A and 600B described herein can be performedsubstantially simultaneously where appropriate or in parallel in someembodiments. Additionally, or alternatively, some portions of process600A and 600B can be omitted in some embodiments. Automatic mappingoperation 600A and 600B can be implemented in any suitable hardwareand/or software. For example, in some embodiments, automatic mappingoperation 600A and 600B can be implemented in automatic mappingmicroscope inspection system 400.

FIG. 8 shows the general configuration of an embodiment of computeranalysis system 450. in accordance with some embodiments of thedisclosed subject matter.

In some embodiments, object identification module 460 can be configuredto receive a low or high resolution scan of a substrate, or a portion ofa substrate, from microscope system 410 and/or any suitable computerreadable media.

Object identification module 460, in some embodiments, can be configuredto detect one or more objects in the received low or high resolutionscan, using image processing algorithms which can include computervision, one or more artificial intelligence algorithm(s) and/or computeralgorithms. Detection of an object can be based on, e.g., a computeraided design (CAD) file of an object, an initial. or earlier objectlayout map of a substrate that is being inspected, images of knownobjects, reference templates for known objects, and/or information aboutknown objects (e.g., an object's dimensions, the mechanical and/orphysical properties of an object).

In some embodiments, object identification module 460 can apply an imageprocessing algorithm, as shown in FIG. 9, to the received substrate scanand. for each object on the substrate or for a region of interest: i)detect the object; ii) determine an object type; iii) determineorientation; iv) identify an imaging alignment position; and/or v)identify a starting scan position. Object identification module 460 canfurther use such information in connection with a reference point on thesubstrate to determine the X, Y, θ coordinates of: i) each object on asubstrate; ii) an imaging alignment position for each object on asubstrate; and/or iii) starting scan position for each object on asubstrate. Object identification module 460 can also be used tocalculate object and/or substrate deformation and/or object shift froman original position (e.g., as compared to an initial object layoutmap).

Detection can refer to visually identifying an object on a substratescan (e.g., by drawing a dashed box around detected objects,) either inprint for a printed substrate scan or visually for a substrate that isdisplayed on a display screen. Object identification module 460, canalso be configured to determine for each detected object additionalinformation including, but not limited to: i) object type; ii) objectorientation: iii) image alignment position; and iv) a starting scanposition. This information can also be displayed visually when asubstrate scan is displayed on a display screen. Alternatively, a textfile can be generated that provides this information.

Object identification module 460 can further be configured to map thedetected objects, imaging alignment position and/or a starting scanposition to an X, Y, θ position in the substrate space in relation toreference markers on the substrate. Further, object identificationmodule 460 can compare each object to a reference image for that objecttype to calculate object/feature deformity. Object identification modulecan also, in some embodiments, calculate an object shift amount bycomparing an object's current X, Y, θ coordinates with an earlier X, Y,θ position, or an expected X, Y, θ position for that object. Note, thatθ or orientation represents the amount an object has rotated about afixed point, compared to an earlier θ position of an object, or anexpected θ position for the object with respect to an origin point forthe substrate, as discussed in connection with FIGS. 10A and 10B.

In some embodiments, an image processing algorithm. based on one or moreimage processing artificial intelligence algorithm(s) can be used todetect objects in the received low or high resolution scans of asubstrate. An image processing algorithm based on artificialintelligence can also be used by object identification module 460 todetermine for each detected object additional information including, butnot limited to: i) object type; ii) object rotation; iii) imagealignment position; and/or iv) a starting scan position. In someembodiments, the algorithm(s) used by object identification module 460can consider context data like location of the object on a substrate,the type of object being inspected, the type of substrate upon which theobject is located, the physical and mechanical properties of theobject/substrate being inspected, similar objects on the same or similartype substrates, a reference template for the inspected object, aninitial object layout map for the inspected substrate to better detectand recognize an object as well as to determine object type, objectrotation, image alignment position, and/or a starting scan position.

An example of an artificial intelligence based image processingalgorithm that can be used by object identification module 460 is imageregistration as described by: Barbara Zitova, “Image RegistrationMethods: A Survey,” Image and Vision Compiling, Oct. 11, 2003, Volume21, Issue 11, pp. 977-1000, which is hereby incorporated by referenceherein in its entirety. The disclosed methods are just examples and arenot intended to be limiting. Further, object identification module 460can use convolutional networks, recurrent neural networks and/or otherartificial neural networks to process the received substrate scans.

In some embodiments, as shown in FIG. 9, an image processing algorithm910 is first trained with training data 920 so that objectidentification module 460 can detect and recognize objects on asubstrate. Training data 920 can include labeled examples of known typesof objects (e.g., the different types of objects that are likely to beinspected on a particular automatic mapping microscope inspection system400). For each type of object, training data 920 can further includelabeled images of actual deformed objects (e.g., objects that havedeformed as a result of the manufacturing process). In furtherembodiments, objects can be artificially deformed according topredefined parameters, and training data 920 can include labeled imagesof such deformed objects. Training data 920 can also include labeledimages of each object type rotated from 0-360 degrees. Further, trainingdata 920 can include labeled images of each type of object to beinspected that identifies a starting scan position and/or imagingalignment position within the image. In some embodiments, training data920 can include data relating to an object's size, shape, composition,location on a substrate, physical/mechanical properties of the objectand/or any other suitable characteristic. In some embodiments, trainingdata can also include unlabeled data.

Once the image processing algorithm is trained it can be applied byobject identification module 460 to a received substrate scan to detectobjects, classify object type, determine object orientation, identify animage alignment position and/or a starting scan position (individuallyand collectively, output data 930).

Object identification module 460 can further be configured to calculateobject deformity, determine an object shift amount, map a detectedobject to X, Y, θ coordinates on a substrate, map an identified imagealignment position for an object to X, Y, θ coordinates on a substrate,map an identified starting scan position for an object to an X, Y, θposition on a substrate and define an imaging sequence based on objecttype and orientation. Each of these functions can be performed with orwithout using artificial intelligence and will be described in greaterdetail in the paragraphs that follow.

In some embodiments, object identification module 460 can calculateobject deformity by comparing deviations in overall dimensions between adetected object and a reference image or between specific features of anobject and a reference image. The reference image can be a referencetemplate for that object type and/or an earlier image of the detectedobject.

Once an object has been detected, object identification module 460, canmap the detected object to a specific X, Y position in a knowncoordinate system, as discussed in connection with FIGS. 3A and 3B.Similarly, object identification module 460 can map an imaging alignmentposition of an object and/or a starting scan position of an object to aspecific X, Y position in a known coordinate system, as discussed inconnection with FIGS. 3A and 3B.

In some embodiments, an object's θ position on the substrate can becalculated using the θ position information output by image processingalgorithm. For example, based on training data of similar objects havinga similar orientation, the image processing algorithm can determine anobject's θ position. In other embodiments, the image processingalgorithm can apply image registration methods to compare an object to areference image and determine the θ position. Some example imageregistration methods to determine rotation are described by BarbaraZitova. “Image Registration Methods: A Survey,” Image and VisionComputing, Oct. 11, 2003, Volume 21, Issue 11, pp. 977-1000, which ishereby incorporated by reference herein in its entirety. In furtherembodiments, object identification module 460 can determine a baselineorientation for each object type using a specific feature or referencepoint, within a reference object. The orientation is with respect to anorigin point of a substrate. To calculate how much the orientation of anobject has changed, object identification module 460 can then compare afeature or reference point for the detected object to a similar featurewithin the reference object using an affine transformation. Affinetransformation is described for example by Donald H. House et al.,Foundations of Physically Based Modeling and Animation, 335-341, 2017,which is hereby incorporated by reference herein in its entirety.

FIGS. 10A and 10B, show an example embodiment where orientation iscalculated based on the rotation of an object A from a first point intime (as represented by FIG. 10A) to a second point in time (asrepresented by FIG. 10B), using the same two reference points on objectA: A₁ and A²). A₁ and A₂ are just examples and other reference points orfeatures on object A can be used. More specifically, as shown in FIG.10A, a first reference point, A₁ for object A can be located at X₁Y₁ ata first point in time using a known coordinate system. A reference line(R₁) can be drawn through reference points A1 and a second referencepoint A2. A first angle θ₁ can be measured based on the intersection ofthe X axis and R₁. As shown in FIG. 10B, the same reference point A₁ forobject A can be located at X₂Y₂ at a second point in time. A referenceline (R₂) can be drawn through reference points A1 and reference pointA2 at their new locations. A second angle θ₂ can be measured based onthe intersection of the X axis and R₂. θ₂ can be subtracted from θ₁ todetermine the orientation of object A in FIG. 10B.

In some embodiments, object identification module 460 can be programmedto associate a predefined imaging sequence for each known object type.When a detected object is classified, object identification module 460can associate the detected object with a predefined imaging sequence forthat object type. Further, object identification module 460 can transmitthis information to control module 440 or to imaging device 420 tocapture images for the detected object applying the predefined imagingsequence.

In some embodiments, object identification module 460 can calculateobject shift amount by comparing a current XY location of an object on asubstrate to an initial or prior XY location of the object (e.g., basedon an initial or prior object layout map). Object identification module460 can transmit this information to object layout map generation module470 and/or object layout prediction module 470.

In some embodiments, object layout prediction module 470 can receivefeedback data and/or object mapping information from objectidentification module 460, along with other context data such as thetype of object being inspected, the type of substrate upon Which theobject is located, the physical and mechanical properties of theobject/substrate being inspected, similar objects on the same or similartype substrates, a reference template for the inspected object, aninitial object layout map for the inspected substrate, etc. The feedbackdata can include, but is not limited to, an X, Y, θ position for eachobject on a substrate at a specific stage in a manufacturing orexamination process, the amount each object on a substrate has deformed,shifted and/or changed its orientation during the manufacturing orexamination process. Object layout prediction module 470 can use thisinformation to make predictions about the X, Y, θ positions of objectsat different stages during the manufacturing or examination processand/or the amount that the objects are likely to deform. Thisinformation can be used to appropriately position objects on a substrateand/or to calibrate steps and/or components in a manufacturing orexamination process to accommodate expected shifting and/or deformationof an object and/or substrate. This information can also be used todetermine if objects on a substrate have moved their position beyond apredicted amount or objects and/or substrate have deformed beyond apredicted amount.

In some embodiments, object layout prediction module can receive aninitial object layout of a substrate and apply a layout predictionalgorithm using artificial intelligence, as shown in FIG. 11 todetermine a new object layout of the substrate at a particular stage ina manufacturing and/or examination process. The new object layout mapcan include for each object on the initial object layout (or any regionof interest) an X, Y, θ position, and/or the amount that the objectsand/or substrate are likely to deform for a particular stage in amanufacturing and/or examination process.

The object layout prediction module can be implemented, in someembodiments, using a linear regression model or a multiple linearregression model. Linear regression modeling is a machine learningtechnique for modeling linear relationships between a dependent variableand one or more independent variables. A simple linear regression modelutilizing a single scalar prediction can be used to perform the objectlayout prediction described herein. Alternatively, a multiple linearregression model utilizing multiple predictors can be used to performthe object layout prediction described herein.

In some embodiments, the object layout prediction algorithm is firsttrained with training data. The training data can include a pair (alsocalled a training example) of input features (X) and output or targetvariables (Y) that the regression learning algorithm is trying topredict. The training examples 1100 can be used to learn a function:hypothesis (H): X→Y, so that H(X) is a reasonable predictor for thecorresponding value of Y. FIG. 11 shows an example training modelaccording to some embodiments of the disclosed subject matter. The inputof training examples 1100 can include, for each object on a substrate: acurrent X, Y, θ position at a first stage in a manufacturing process(e.g., initial layout map, photoresist step, cleaning step, pre-dicingstep) and an object/substrate type. The output of training examples 1100can include for each object on a substrate an X, Y, θ position,deformity, and/or shift amount at a second stage in a manufacturingprocess. Once trained, object layout algorithm 1110, can receive anobject layout map for a substrate at a first manufacturing and/orexamination instance, as well as other information aboutobject/substrate type, and predict X, Y, θ positions of the objects onthe substrate and/or amount of deformity to expect for the objectsand/or substrate at a second instance of a manufacturing or examinationprocess. Object layout algorithm 1110 can continuously or periodicallyreceive feedback data from object identification module 460 and modifyhypothesis (H).

As explained in connection with 630B of FIG. 6B, object layoutprediction module 470 can compare the predicted position of objects on asubstrate to the actual position information generated by objectidentification module 460 and determine whether to generate an alert.Further, object layout prediction module 470 can compare the predictedobject position to the actual position information generated by objectidentification module 460 and/or compare the predicted object deformityto the actual object deformity information generated by objectidentification module 460 to assess the accuracy of the predictions ofobject layout prediction. module 470.

The functionality of the components for automatic mapping microscopeinspection system 400 can be combined into a single component or spreadacross several components. In some embodiments, the functionality ofsome of the components (e.g., computer processing by computer analysissystem 450) can be performed remotely from microscope. system 410. Insome embodiments, control analysis system can be combined intomicroscope system 410.

Note that automatic mapping microscope inspection system 400 can includeother suitable components not shown. Additionally or alternatively, someof the components included in automatic mapping microscope inspectionsystem 400 can be omitted.

In some embodiments, any suitable computer readable media can be usedfor storing instructions for performing the functions and/or processesdescribed herein. For example, in some embodiments, computer readablemedia can be transitory or non-transitory. For example, non-transitorycomputer readable media can include media such as non-transitorymagnetic media (such as hard disks, floppy disks, etc.), non-transitoryoptical media (such as compact discs, digital video discs, Blu-raydiscs, etc.), non-transitory semiconductor media (such as flash memory,electrically programmable read only memory (EPROM), electricallyerasable programmable read only memory (EEPROM), etc.), any suitablemedia that is not fleeting or devoid of any semblance of permanenceduring transmission, and/or any suitable tangible media.

As another example. transitory computer readable media can includesignals on networks, in wires, conductors, optical fibers, circuits, andany suitable media that is fleeting and devoid of any semblance ofpermanence during transmission, and/or any suitable intangible media.

The various systems, methods, and computer readable mediums describedherein can be implemented as part of a cloud network environment. Asused in this paper, a cloud-based computing system is a system thatprovides virtualized computing resources, software and/or information toclient devices. The computing resources, software and/or information canbe virtualized by maintaining centralized services and resources thatthe edge devices can access over a communication interface, such as anetwork. The cloud can provide various cloud computing services viacloud elements, such as software as a service (SaaS) (e.g.,collaboration services, email services, enterprise resource planningservices, content services, communication services, etc.),infrastructure as a service (IaaS) (e.g., security services, networkingservices, systems management services, etc.), platform as a service(PaaS) (e.g., web services, streaming services, application developmentservices, etc.), and other types of services such as desktop as aservice (DaaS), information technology management as a service (ITaaS),managed software as a service (MSaaS), mobile backend as a service(MBaaS), etc.

The provision of the examples described herein (as well as clausesphrased as “such as,” “e.g.,” “including,” and the like) should not beinterpreted as limiting the claimed subject matter to the specificexamples; rather, the examples are intended to illustrate only some ofmany possible aspects. A person of ordinary skill in the art wouldunderstand that the term mechanism can encompass hardware, software,firmware, or any suitable combination thereof.

Unless specifically stated otherwise as apparent from the abovediscussion, it is appreciated that throughout the description,discussions utilizing terms such as “determining,” “providing,”“identifying,” “comparing” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system memories or registersor other such information storage, transmission or display devices.

Certain aspects of the present disclosure include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the present disclosurecould be embodied in software, firmware or hardware, and when embodiedin software, could be downloaded to reside on and be operated fromdifferent platforms used by real time network operating systems.

The present disclosure also relates to an apparatus for perform g theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored on acomputer readable medium that can be accessed by the computer. Such acomputer program may be stored in a computer readable storage mediumsuch as, but is not limited to, any type of disk including floppy disks,optical disks, CD-ROMs, magnetic-optical disks, read-only memories(ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic oroptical cards, application specific integrated circuits (ASICs), or anytype of non-transient computer-readable storage medium suitable forstoring electronic instructions. Furthermore, the computers referred toin the specification may include a single processor or may bearchitectures employing multiple processor designs for increasedcomputing capability.

The algorithms and operations presented herein are not inherentlyrelated to any particular computer or other apparatus. Variousgeneral-purpose systems may also be used with programs in accordancewith the teachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps andsystem-related actions. The required structure for a variety of thesesystems will be apparent to those of skill in the art, along withequivalent variations. In addition, the present disclosure is notdescribed with reference to any particular programming language. It isappreciated that a variety of programming languages may be used toimplement the teachings of the present disclosure as described herein,and any references to specific languages are provided for disclosure ofenablement and best mode of the present disclosure.

The automatic mapping of fluid objects on a substrate mechanism, methodand system have been described in detail with specific reference tothese illustrated embodiments. It will be apparent, however, thatvarious modifications and changes can be made within the spirit andscope of the disclosure as described in the foregoing specification, andsuch modifications and changes are to be considered equivalents and partof this disclosure. The scope of the present disclosure is limited onlyby the claims that follow.

1. A method, comprising: generating, by a computing system, a predictionmodel for predicting coordinates of an object on a substrate at eachstage of processing by: generating a training data set comprisinglabeled examples of known objects on substrates and labeled examples ofdeformed objects on substrates, and training the prediction model todetect objects on the substrates and predict coordinates of each objectat each stage of processing; receiving, by the computing system, a scanof a set of target objects positioned on a stage of a microscope system;identifying, by the computing system, a position of each target objectin the set of target objects; and predicting, by the computing systemvia the prediction model, coordinates of each target object in the setof target objects at different stages of examination.
 2. The method ofclaim 1, wherein predicting, by the computing system, the coordinates ofeach target object in the set of target objects at different stages ofexamination comprises: predicting a second set of coordinates of eachtarget object at a downstream process.
 3. The method of claim 1, furthercomprising: comparing, by the computing system, the predictedcoordinates of each target object in the set of target objects to theidentified position of each target object in the set of target objects;and generating, by the computing system, an alert upon determining thatthe predicted coordinates of each target object exceeds a thresholdtolerance from the identified position of each target object.
 4. Themethod of claim 1, further comprising: assessing, by the computingsystem, a degree of deformity of each target object based on thepredicting.
 5. The method of claim 4, further comprising: identifying,by the computing system, an actual degree of deformity of each targetobject following a downstream processing stage; and assessing, by thecomputing system, an accuracy of the prediction model based on acomparison between the actual degree of deformity and the degree ofdeformity.
 6. The method of claim 1, further comprising: generating, bythe computing system, object mapping information based on thepredicting.
 7. The method of claim 6, further comprising: generating, bythe computing system, an object layout map using the object mappinginformation.
 8. A microscope inspection system, comprising: one or moreprocessors in communication with a microscope system that includes astage for moving a set of objects disposed on the stage and an imagingdevice, wherein the imaging device scans the set of objects from thestage; and a memory having programming instructions stored thereon,which, when executed by the one or more processors, performs one or moreoperations, comprising: generating a prediction model for predictingcoordinates of an object on a substrate at each stage of processing by:generating a training data set comprising labeled examples of knownobjects on substrates and labeled examples of deformed objects onsubstrates, and training the prediction model to detect objects on thesubstrates and predict coordinates of each object at each stage ofprocessing; receiving a scan of a set of target objects positioned onthe stage of the microscope system; identifying a position of eachtarget object in the set of target objects; and predicting, by theprediction model, coordinates of each target object in the set of targetobjects at different stages of examination.
 9. The microscope inspectionsystem of claim 8, wherein predicting, by the prediction model, thecoordinates of each target object in the set of target objects atdifferent stages of examination comprises: predicting a second set ofcoordinates of each target object at a downstream process.
 10. Themicroscope inspection system of claim 8, wherein the one or moreoperations further comprise: comparing the predicted coordinates of eachtarget object in the set of target objects to the identified position ofeach target object in the set of target objects; and generating an alertupon determining that the predicted coordinates of each target objectexceeds a threshold tolerance from the identified position of eachtarget object.
 11. The microscope inspection system of claim 8, whereinthe one or more operations further comprise: assessing a degree ofdeformity of each target object based on the predicting.
 12. Themicroscope inspection system of claim 11, wherein the one or moreoperations further comprise: identifying an actual degree of deformityof each target object following a downstream processing stage; andassessing an accuracy of the prediction model based on a comparisonbetween the actual degree of deformity and the degree of deformity. 13.The microscope inspection system of claim 8, wherein the one or moreoperations further comprise: generating object mapping information basedon the predicting.
 14. The microscope inspection system of claim 13,wherein the one or more operations further comprise: generating anobject layout map using the object mapping information.
 15. A method,comprising: training, by a computing system, a prediction model topredict coordinates of an object on a substrate at a downstreamprocessing stage based on detected positions of objects on the substrateat an upstream processing stage; receiving, by the computing system, ascan of a set of target objects positioned on a stage of a microscopesystem; identifying, by the computing system, a position of each targetobject in the set of target objects; and predicting, by the computingsystem via the prediction model, coordinates of each target object inthe set of target objects at different stages of examination.
 16. Themethod of claim 15, further comprising: comparing, by the computingsystem, the predicted coordinates of each target object in the set oftarget objects to the identified position of each target object in theset of target objects; and generating, by the computing system, an alertupon determining that the predicted coordinates of each target objectexceeds a threshold tolerance from the identified position of eachtarget object.
 17. The method of claim 15, further comprising:assessing, by the computing system, a degree of deformity of each targetobject based on the predicting.
 18. The method of claim 17, furthercomprising: identifying, by the computing system, an actual degree ofdeformity of each target object following a downstream processing stage;and assessing, by the computing system, an accuracy of the predictionmodel based on a comparison between the actual degree of deformity andthe degree of deformity.
 19. The method of claim 15, further comprising:generating, by the computing system, object mapping information based onthe predicting.
 20. The method of claim 19, further comprising:generating, by the computing system, an object layout map using theobject mapping information.